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Executive Summary 



The District of Columbia School Choice Incentive Act of 2003, passed by Congress in 
January 2004, established the first federally funded, private school voucher program in the United States. 
As part of this legislation. Congress mandated a rigorous evaluation of the impacts of the Program, now 
called the DC Opportunity Scholarship Program (OSP). This report presents findings from the evaluation 
of the impacts 3 years after families who applied were given the option to move from a public school to a 
participating private school of their choice. 



The evaluation is based on a randomized controlled trial design that compares the outcomes 
of eligible applicants randomly assigned to receive (treatment group) or not receive (control group) a 
scholarship through a series of lotteries. The main findings of the evaluation so far include: 

• After 3 years, there was a statistically significant positive impact on reading test 
scores, but not math test scores. Overall, those offered a scholarship were performing 
at statistically higher levels in reading — equivalent to 3.1 months of additional 
learning — but at similar levels in math compared to students not offered a scholarship 
(table 3). Analysis in prior years indicated no significant impacts overall on either 
reading or math achievement. 

• The OSP had a positive impact overall on parents’ reports of school satisfaction 
and safety (figures 3 and 4), but not on students’ reports (figures 3 and 4). Parents 
were more satisfied with their child’s school (as measured by the percentage giving the 
school a grade of A or B) and viewed their child’s school as safer and more orderly if 
the child was offered a scholarship. Students had a different view of their schools than 
did their parents. Reports of safety and school climate were comparable for students in 
the treatment and control groups. Overall, student satisfaction was unaffected by the 
Program. 

• This same pattern of findings holds when the analysis is conducted to determine 
the impact of using a scholarship rather than being offered a scholarship. Fourteen 
percent of students in our impact sample who were randomly assigned by lottery to 
receive a scholarship and who responded to year 3 data collection chose not to use their 
scholarship at any point over the 3-year period after applying to the Program.' We use 
a common statistical technique to take those “never users” into account; it assumes that 
the students had zero impact from the OSP, but it does not change the statistical 
significance of the original impact estimates. Therefore, the positive impacts on 
reading achievement, parent views of school safety and climate, and parent views of 



' This 14 percent “never user” rate among year 3 respondents in the impact sample differs from the 25 percent “never user” rate for the impact 
sample as a whole (Figure 1) because scholarship “never users” in the impact sample responded to year 3 data collection events at lower rates 
than did scholarship “ever users.” 
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satisfaction all increase in size, and there remains no impact on math achievement and 
no overall impact on students’ perceptions of school safety and climate or satisfaction 
from using an OSP scholarship. 

• The OSP improved reading achievement for 5 of the 10 subgroups examined.^ 

Being offered or using a scholarship led to higher reading test scores for participants 
who applied from schools that were not classified as “schools in need of improvement” 
(non-SINI). There were also positive impacts for students who applied to the Program 
with relatively higher levels of academic performance, female students, students 
entering grades K-8 at the time of application, and students from the first cohort of 
applicants. These impacts translate into 1/3 to 2 years of additional learning growth. 
However, the positive subgroup reading impacts for female students and the first 
cohort of applicants should be interpreted with caution, as reliability tests suggest that 
they could be false discoveries. 

• No achievement impacts were observed for five other subgroups of students, 
including those who entered the Program with relative academic disadvantage. 

Subgroups of students who applied from SINI schools (designated by Congress as the 
highest priority group for the Program) or were in the lower third of the test score 
distribution among applicants did not demonstrate significant impacts on reading test 
scores if they were offered or used a scholarship. In addition, male students, those 
entering high school grades upon application, and those in application cohort 2 showed 
no significant impacts in either reading or math after 3 years. 



DC Opportunity Scholarship Program 

The purpose of the new scholarship program was to provide low-income residents, 
particularly those whose children attend schools in need of improvement or corrective action under the 
Elementary and Secondary Education Act, with “expanded opportunities to attend higher performing 
schools in the District of Columbia” (Sec. 303). The scholarship, worth up to $7,500, could be used to 
cover the costs of tuition, school fees, and transportation to a participating private school. The statute also 
prescribed how scholarships would be awarded: (1) in a given year, if there are more eligible applicants 
than available scholarships or open slots in private schools, scholarships are to be awarded by random 
selection (e.g., by lottery), and (2) priority for scholarships is given first to students attending SINI public 
schools and then to families that lack the resources to take advantage of school choice options. 



^ The subgroups that are analyzed in this study were designated prior to the colleetion and analysis of data and are of particular policy interest 
based on the Program statute and education policy literature. The subgroups are: (1) whether students attended a school designated as in need 
of improvement (SINI) under the No Child Left Behind Act prior to application to the Program — students were either attending a SINI-ever or 
SINI-never school; (2) whether students were relatively lower performing or relatively higher performing at baseline — students were either in 
the bottom one-third or the top two-thirds of the test score distribution; (3) student gender; (4) whether students were entering grades K-8 or 9- 
12 at the time of application; and (5) whether students were in application cohort 1 (applied in 2004) or application cohort 2 (applied in 2005). 



VI 




The Program is operated by the Washington Scholarship Fund (WSF). To date, there have 
been five rounds of applications to the OSP (table 1). Applicants in spring 2004 (cohort 1) and spring 
2005 (cohort 2) represent the majority of Program applicants; the evaluation sample was drawn from 
these two groups.^ A smaller number of applicants in spring 2006 (cohort 3), spring 2007 (cohort 4), and 
spring 2008 (cohort 5) were recruited and enrolled by WSF in order to keep the Program operating at 
capacity each year. 

Table 1. OSP Applicants by Program Status, Cohorts 1 Through 5, Years 2004-2008 





Cohort 1 Cohort 2 

(Spring 2004) (Spring 2005) 


Total 

Cohort 1 and 
Cohort 2 


Cohort 3 
(Spring 2006), 
Cohort 4 
(Spring 2007), 
and Cohort 5 
(Spring 2008) 


Total, All 
Cohorts 


Applicants 


2,692 


3,126 


5,818 


2,034 


7,852 


Eligible applicants 


1,848 


2,199 


4,047 


1,284 


5,331 


Scholarship awardees 


1,366 


1,088 


2,454 


1,284 


3,738 


Scholarship users in initial year of receipt 


1,027 


797 


1,824 


1,057 


2,881 


Scholarship users fall 2005 


919 


797 


1,716 


NA 


1,716 


Scholarship users fall 2006 


788 


684 


1,472 


333 


1,805 


Scholarship users fall 2007 


678 


581 


1,259 


671 


1,930 


Scholarship users fall 2008 


496 


411 


909 


807 


1,714 



NOTES: Because most participating private schools closed their enrollments by mid-spring, applicants generally had their eligibility 

determined based on income and residency, and the lotteries were held prior to the administration of baseline tests. Therefore, 
baseline testing was not a condition of eligibility for most applicants. The exception was applicants entering the highly 
oversubscribed grades 6-12 in cohort 2. Those who did not participate in baseline testing were deemed ineligible for the lottery and 
were not included in the eligible applicant figure presented above, though they were counted in the applicant total. In other words, 
the cohort 2 applicants in grades 6-12 had to satisfy income, residency, and baseline testing requirements before they were 
designated eligible applicants and entered in the lottery. 

The initial year of scholarship receipt was fall 2004 for cohort 1, fall 2005 for cohort 2, fall 2006 for cohort 3, fall 2007 for cohort 
4, and fall 2008 for cohort 5. 

SOURCES: OSP applications and WSF’s enrollment and payment files. 



Mandated Evaluation of the OSP 



In addition to establishing the OSP, Congress mandated an independent evaluation of it be 
conducted, with annual reports on the progress of the study. The legislation indicated the evaluation 
should analyze the effects of the Program on various academic and nonacademic outcomes of concern to 
policymakers and use “. . . the strongest possible research design for determining the effectiveness” of the 



^ Descriptive reports on each of the first 2 years of implementation and cohorts of students have been previously prepared and released (Wolf, 
Gutmann, Eissa, Puma, and Silverberg 2005; Wolf, Gutmann, Puma, and Silverberg 2006) and are available on the Institute of Education 
Sciences’ website at http://ies.ed.gov/ncee. 
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Program. The current evaluation was developed to be responsive to these requirements. In particular, the 
foundation of the evaluation is a randomized controlled trial (RCT) that compares outcomes of eligible 
applicants (students and their parents) randomly assigned to receive or not receive a scholarship. This 
decision was based on the mandate to use rigorous evaluation methods, the expectation that there would 
be more applicants than funds and private school spaces available, and the statute’s requirement that 
random selection be the vehicle for determining who receives a scholarship. An RCT design is widely 
viewed as the best method for identifying the independent effect of programs on subsequent outcomes 
(e.g., Boruch, de Moya, and Snyder 2002, p. 74). Random assignment has been used by researchers 
conducting impact evaluations of other scholarship programs in Charlotte, NC; New York City; Dayton, 
OH; and Washington, DC (Greene 2001; Howell et al. 2002; Mayer et al. 2002). 

The recruitment, application, and lottery process conducted by WSF with guidance from the 
evaluation team created the foundation for the evaluation’s randomized trial and determined the group of 
students for whom impacts of the Program are analyzed in this report. Because the goal of the evaluation 
was to assess both the short-term and longer term impacts of the Program, it was necessary to focus the 
study on early applicants to the Program (cohorts 1 and 2) whose outcomes could be tracked over at least 
3 years during the evaluation period. During the first 2 years of recruitment, WSF received applications 
from 5,818 students. Of these, approximately 70 percent (4,047 of 5,818) were eligible to enter the 
Program (table 1). Of the total pool of eligible applicants, 2,308 students who were attending public 
schools or were rising kindergarteners entered lotteries (492 in cohort 1; 1,816 in cohort 2), resulting in 
1,387 students assigned to the treatment condition and 921 assigned to the control condition. These 
students constitute the evaluation’s impact analysis sample and represent three-quarters of all students in 
cohorts 1 and 2 who were not already attending a private school when they applied to the OSP. 

Data are collected from the impact sample each year, starting with the spring in which 
students applied to the OSP (baseline) and each spring thereafter. These data include assessments of 
student achievement in reading and mathematics using the Stanford Achievement Test version 9 (SAT- 
9),"* surveys of parents, and surveys of students in grade 4 and above — administered by the evaluation 
team in central District of Columbia (DC) locations on Saturdays or weekday evenings because neither 
the public nor private schools would allow data collection on their campuses during the school day. In 
addition, the evaluation surveys all DC public and private schools each spring in order to address the 
statute’s interest in understanding how the schools are responding to the OSP. 



Stanford Abbreviated Achievement Test (Form S), Ninth Edition. San Antonio, TX: Harcourt Educational Measurement, Harcourt Assessment, 
Inc., 1997. 
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Participation in the OSP 



In interpreting the impacts of the OSP, it is useful to examine the characteristics of the 
private schools that participate in the Program and the extent to which students offered scholarships (the 
treatment group) moved into and out of them during the first 3 years. 



School Participation 



The private schools participating in the OSP represent the choice set available to parents 
whose children received scholarships. That group of schools had mostly stabilized by the 2005-06 school 
year. The schools that offered the most slots to OSP students, and in which OSP students and the impact 
sample’s treatment group were clustered, have characteristics that differed somewhat from the average 
participating OSP school. Although 56 percent of all participating schools were faith-based (39 percent 
were part of the Catholic Archdiocese of Washington), 82 percent of the treatment group attended a faith- 
based school, with 59 percent of them attending the 22 participating Catholic parochial schools (table 2). 
Twenty-two percent of treatment group students were attending a school that charged tuition above the 
statutory cap of $7,500 during their third year in the Program (table 2) even though 38 percent and 46 
percent of participating schools charged tuitions above that cap in 2006-07 and 2007-08, respectively. 

Table 2. Features of Participating OSP Private Schools Attended by the Treatment Group in 
Year 3 



Characteristic 


Weighted 

Mean 


Highest 


Lowest 


Valid V 


Archdiocesan Catholic schools (percent 
of treatment students attending) 


59.2 


NA 


NA 


66 


Other faith-based schools (percent of 
treatment students attending) 


22.5 


NA 


NA 


66 


Charging over $7,500 tuition (percent of 
treatment students attending) 


22.3 


NA 


NA 


48 


Tuition 


$6,620 


$29,902 


$3,600 


48 


Enrollment 


260.5 


1,072 


10 


43 


Student N 


701 









NOTES: “Valid V” refers to the number of schools for which information on a particular characteristic was available. When a tuition range was 

provided, the mid-point of the range was used. The weighted mean was generated by associating each student with the characteristics 
of the school he/she was attending, and then computing the average of these student-level characteristics. 

SOURCE: OSP School Directory information, 2004-05, 2005-06, 2006-07, and 2007-08, Washington Scholarship Fund. 



While the characteristics of the participating private schools are important considerations for 
parents, in many respects it is how the schools differ from the public school options available to them that 
matters most. In the third year after applying to the OSP, students in the treatment and control groups did 
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not differ significantly regarding the proportion attending schools that offered a separate library (88 vs. 91 
percent), gyms (71 and 72 percent), and art programs (89 and 87 percent). There were the following 
statistically significant differences (at the .01 level): 



• Students in the treatment group were more likely than those in the control group to 
attend schools with a computer lab (96 vs. 87 percent), with special programs for 
advanced learners (48 vs. 32 percent), and that offered a music program (89 vs. 82 
percent). 

• Students in the treatment group were less likely than the control group to attend a 
school with a cafeteria facility (79 vs. 88 percent) or a nurse’s office (30 vs. 81 
percent). 

• Students in the treatment group were also less likely than those in the control group to 
attend a school that offered special programs for non-English speakers (26 vs. 57 
percent), special programs for students with learning problems (71 vs. 88 percent), 
counselors (69 vs. 82 percent), tutors (50 vs. 67 percent), and after-school programs 
(86 vs. 92 percent). 

Student Participation 



As has been true in similar programs, not all students offered an OSP scholarship actually 
used it to enroll in a private school. For students assigned to the treatment group, during the first 3 years 
of the Program (figure 1): 

• 25 percent (346 out of 1,387 students) of those offered an OSP scholarship never used 
it; 

• 34 percent (473 students) used their scholarship during some but not all of the first 3 
years after the award; and 

• The remaining 41 percent (568 students) used their scholarship consistently for the 
entire 3 years after the lottery. 

The reasons for not using the scholarship — either initially or consistently — ^varied. The most 
common reasons cited by parents whose child never used their scholarship at anytime in year 3 and who 
completed surveys were (figure 2): 

• Lack of available space in the private school they wanted their child to attend (22 
percent of these parents); 

• Child moved out of DC (21 percent of these parents); 

• Child was accepted into a public charter school (19 percent of these parents); and 
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• Participating schools did not offer services for their child’s learning or physical 
disability or other special needs (16 percent of these parents). 

Figure 1. Proportions of Treatment Group Students Who Experienced 
Various Categories of Usage in First 3 Years 




NOTES: Data are not weighted. Valid N = 1,387. Students were identified as scholarship users based upon 

information from WSF’s payment files. Because some schools use a range of tuitions and some 
students had alternative sources of funding, students were classified as full users if WSF made 
payments on their behalf that equaled at least 80 percent of the school’s annual tuition. Otherwise, 
students were identified as partial users (1 percent to 79 percent of tuition paid) or nonusers (no 
payments). 

SOURCES: OSP applications and WSF’s payment files. 



The most common responses given by parents whose child initially used a scholarship in year 3 but 
dropped out of the OSP include: 

• Lack of academic support that the child needed (39 percent of these parents); 

• "Child did not like the private school" (25 percent); 

• There was another private school the child liked better (13 percent); 

• Work at the private school was too hard (1 1 percent); 

• It was too difficult to get the child to the private school each day (1 1 percent); and 

• The discipline or rules at the private school were too strict (7 percent). 
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Figure 2. 



Most Common Reasons Given by Parents for Declining to Use 
the OSP Scholarship in Year 3 




NOTES: Responses are unweighted. Respondents were able to select multiple responses, which generated a 

total of 180 responses provided by 153 parents. This equates to an average of 1.2 responses per 
parent. 

SOURCE: Impact Evaluation Parent Surveys. 



Students who were partial users were more likely to have special needs and those entering the higher 
grades averaged lower baseline test scores than students who participated consistently across the 3 years. ^ 

Students who never used the OSP scholarship offered to them, or who did not use the 
scholarship consistently, could have found their way into other (non-OSP-participating) private schools, 
public charter schools, or traditional DC public schools. The same alternatives were available to students 
who applied to the OSP but were never offered a scholarship (the impact sample’s control group). Both 
the treatment and control groups moved between public (both traditional and charter) and private schools 
or between SINI and non-SINI schools. As a result, over the 3 years after they applied to the OSP: 

• Among the treatment group, 3 percent remained in the same school they were in when 
they applied to the Program; 46 percent switched schools once; 40 percent switched 
schools twice; and 1 1 percent switched three times. 



^ At baseline, partial users in grades 9-12 were lower performing in reading (27 National Percentile Ranks (NPRs) vs. 40 NPRs for full users, 
statistically significant at the .05 level) and in math (29 NPRs vs. 49 NPRs for full users, statistically significant at the .01 level); partial users 
in grades 6-8 were lower performing in math (34 NPRs vs. 41 NPRs for full users, statistically significant at the .01 level); and partial users 
were more likely to have special needs (5 percent vs. 10 percent for full users, statistically significant at the .05 level). 
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• Among the control group, 1 5 percent remained in the same school they were in when 
they applied to the Program; 40 percent switched schools once; 37 percent switched 
schools twice; and 8 percent switched three times. 

These patterns of student mobility are important because previous studies suggest that 
switching schools has an initial short-term negative effect on student achievement (Hanushek, Kain, and 
Rivkin 2004). 



Impact of the Program After 3 Years: Key Outcomes 

The statute that authorized the OSP mandated that the Program be evaluated with regard to 
its impact on student test scores and school safety, as well as the “success” of the Program, which, in the 
design of this study, includes satisfaction with school choices. The impacts of the Program on these 
outcomes are presented in two ways: (1) the impact of the offer of an OSP scholarship, derived straight 
from comparing outcomes of the treatment and control groups, and (2) the impact of using an OSP 
scholarship, calculated from the unbiased treatment-control group comparison, but statistically adjusting 
for students who declined to use their scholarships.^ The main focus of this study was on the overall 
group of students, with a secondary interest in students who applied from SINI schools, followed by other 
subgroups of students (e.g., defined by their academic performance at application, their gender, or their 
grade level). 



Previous reports released in spring 2007 and spring 2008 indicated that 1 and 2 years after 
application, there were no statistically significant impacts on overall academic achievement or on student 
perceptions of school safety or satisfaction (Wolf et al. 2007; Wolf et al. 2008). Parents were more 
satisfied if their child was in the Program and viewed their child’s school as safer and more orderly. 
Among the secondary analyses of subgroups, there were impacts on math test scores in year 1 for students 
who applied from non-SINI schools and those with relatively higher pre-Program test scores, and impacts 
in reading test scores (but not math) in year 2 for those same two subgroups plus students who applied in 
the first year of Program implementation. However, these findings were no longer statistically significant 
when subjected to a reliability test to adjust for the multiple comparisons of treatment and control group 
students across 10 subgroups; the results may be “false discoveries” and should therefore be interpreted 
and used with caution. Throughout this report, the phrases “appears to have an impact” and “may have 



’’ This analysis uses straightforward statistical adjustments to account not only for the approximately 14 percent of impact sample year 3 
respondents who received the offer of a scholarship but declined to use it over the 3-year period after application (the “never users”), but also 
the estimated 1.6 percent of the control group who never received a scholarship offer but who, by virtue of having a sibling with an OSP 
scholarship, ended up in a participating private school (we call this “program-enabled crossover”). These adjustments increase the size of the 
scholarship offer effect estimates, but do not alter the statistical significance of the impact estimate. 
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had an impact” are used to caution readers regarding statistically significant impacts that may have been 
false discoveries. 



The analyses in this report were conducted using data collected on students 3 years after they 
applied to the OSP.’ 



Impacts on Students and Parents Overall 

• Across the full sample, there was a statistically significant impact on reading 
achievement of 4.5 scale score points (effect size (ES) = .13)* from the offer of a 
scholarship and 5.3 scale score points (ES = .15) from the use of a scholarship 
(table 3). These impacts are equivalent to 3.1 and 3.7 months of additional learning, 
respectively.^ 

• There was no statistically significant impact on math achievement, overall (ES = .03) 
from the offer of a scholarship nor from the use of a scholarship (table 3).'° 

• Parents of students offered a scholarship were more likely to report their child’s school 
to be safer and have a more orderly school climate (ES = .29) compared to parents of 
students not offered a scholarship (figure 3); the same was true for parents of students 
who chose to use their scholarships (ES = .34). 

• On the other hand, students who were offered a scholarship reported similar levels of 
school safety and an orderly climate compared to those in the control group (ES = .06; 
figure 3); there was also no significant impact on student reports of school safety and 
an orderly climate from using a scholarship (ES = .07). 

• The Program produced a positive impact on parent satisfaction with their child’s school 
as measured by the likelihood of grading the school an “A” or “B,” both for the impact 
of a scholarship offer (ES = .22; figure 4) and the impact of scholarship use (ES = .26). 



^ Specifically, year 3 test scores were obtained from 69 percent of study participants, whereas parent survey data were gathered from 68 percent 
of participants and student survey data from 67 percent of participants. Response rates to the principal survey varied between 51.8 percent and 
57.3 percent, depending on academic year and school sector. Missing outcome data create the potential for nonresponse bias in a longitudinal 
evaluation such as this one, if the nonrespondent portions of the sample are different between the treatment and control groups. Response rates 
differed by less than 2 percent between the treatment and control groups for the tests and parent and student surveys, meaning that similar 
proportions of the treatment and control groups provided outcome data. In addition, nonresponse weights were used to equate the two groups 
on important baseline characteristics, thereby reducing the threat of nonresponse bias in this case. 

^ An effect size (ES) is a standardized measure of the relative size of a program impact. In this report, effect sizes are expressed as a proportion 
of a standard deviation of the distribution of values observed for the study control group. One full standard deviation above and below the 
average value for a variable such as outcome test scores contains 64 percent of the observations in the distribution. Two full standard 
deviations above and below the average contain 95 percent of the observations. 

^ Scale score impacts were converted to approximate months of learning first by dividing the impact ES by the ES of the weighted (by grade) 
average annual increase in reading scale scores for the control group. The result was the proportion of a typical year of achievement gain 
represented by the programmatic impact. That number was further divided by nine to convert the magnitude of the gain to months, since the 
official school year in the District of Columbia comprises 9 months of instruction. 

The magnitudes of these estimated achievement effects are below the threshold of .12 standard deviations, estimated by the power analysis to 
be the study’s Minimum Detectable Effect (MDE) size. 
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Table 3. 



Year 3 Impact Estimates of the Offer and Use of a Scholarship on the Full Sample: 
Academic Achievement 



Impact of the Scholarship Offer (ITT) 


Impact of Scholarship 
Use (lOT) 






Treatment 


Control 


Difference 




Adjusted 






Student 


Group 


Group 


(Estimated 




Impact 




7 J-value of 


Achievement 


Mean 


Mean 


Impact) 


Effect Size 


Estimate 


Effect Size 


estimates 


Reading 


635.44 


630.98 


4.46* 


.13 


5.27* 


.15 


.01 


Math 


630.15 


629.35 


.81 


.03 


.95 


.03 


.62 



* Statistically significant at the 95 percent confidence level. 

NOTES: Means are regression adjusted using a consistent set of baseline covariates. Impacts are displayed in terms of scale scores. Effect sizes 

are in terms of standard deviations. Valid N for reading = 1,460; math = 1,468. Separate reading and math sample weights used. 



Figure 3. Parent Perceptions and Student Reports of Safety and an 
Orderly School Climate 



Year 3 Group Means: Parent Year 3 Group Means: Student 

Perceptions Reports 





**Statistically significant at the 99 percent confidence level. 

NOTES: Parent perceptions are based on a ten-point scale; student reports are based on an eight-point scale. For 

parent perceptions, valid N = 1,423; parent survey weights were used; the ten-point index of indicators 
of school safety and an orderly environment includes the absence of property destruction, tardiness, 
truancy, fighting, cheating, racial conflict, weapons, drug distribution, drug/alcohol use, and teacher 
absenteeism. For student reports, valid N = 1,098; student survey weights were used; the survey was 
given to students in grades 4-12; the means represent the absence of incidents on an eight-item index 
for student reports of students being a victim of theft, drug-dealing, assaults, threats, bullying or 
taunting, or had observed weapons at school. Means are regression adjusted using a consistent set of 
baseline covariates. 
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Figure 4. 



Parent and Student Reports of School Satisfaction 



Year 3 Group Means: Percentage Year 3 Group Means: Percentage 

of Parents Who Gave School of Students Who Gave School 

Grade A or B Grade A or B 




** Statistically significant at the 99 percent confidence level. 

NOTES: For parent reports, valid N = 1,410; parent survey weights were used. For student reports, valid N = 

1,014; student survey weights were used; the survey was given to students in grades 4-12. Means are 
regression adjusted using a consistent set of baseline covariates. 



• Overall, there were no impacts of the OSP from being offered (ES = -.06; figure 4) or 
using a scholarship (ES = -.07) on students’ satisfaction with their schools as measured 
by the likelihood of assigning their school a grade of “A” or “B.” 



Impacts on Subgroups 

In addition to determining the general impacts of the OSP on all study participants, this 
evaluation also reports programmatic impacts on policy-relevant subgroups of students. The subgroups 
were designated prior to data collection and include students who were attending SINI versus non-SINI 
schools at application, those relatively higher or lower performing at baseline, girls or boys, elementary 
versus high school students, and those from application cohort 1 or cohort 2. Since the subgroup analysis 
involves significance tests across multiple comparisons of treatment and control students, some of which 
may be statistically significant merely by chance, these subgroup-specific results should be interpreted 
with caution. Specifically: 
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Subgroup Achievement Impacts 



• There were no statistically significant reading (ES = .05) or math (ES = .01) 
achievement impacts for the high-priority subgroup of students who had attended a 
SINI public school under No Child Left Behind (NCLB) before applying to the 
Program. 

• There were statistically significant impacts on reading test scores in year 3 for five 
subgroups of students, although the statistical significance of two of the subgroup 
findings was not robust to adjustments for multiple comparisons: 

o Students who attended non-SINI public schools prior to application to the 
Program (56 percent of the impact sample) scored an average of 6.6 scale score 
points higher in reading (ES = .19) if they were offered the scholarship 
compared to not being offered a scholarship and 7.7 scale score points higher 
(ES = .22) if they used their scholarship compared to not being offered a 
scholarship. These scale score differences between the treatment and control 
groups translate into 4. 1 and 4.9 additional months of learning, or half a year of 
schooling based on a typical 9-month school year. 

o Students who entered the Program in the higher two-thirds of the test-score 
performance distribution at baseline (66 percent of the impact sample) scored an 
average of 5.5 scale score points higher in reading (ES = .17) if they were 
offered a scholarship and 6.2 scale score points higher (ES = .19) if they used 
their scholarship, impacts equivalent to 4.0 and 4.6 months of learning gains. 

o Female students scored an average of 5. 1 scale score points higher in reading 
(ES = .15) if they were offered a scholarship and 5.8 scale score points higher 
(ES = .17) if they used their scholarship. These impacts represent 3.1 and 3.6 
months of additional learning, respectively. The statistical significance of this 
finding was not robust to adjustments for multiple comparisons. 

o Students who entered the Program in grades K-8 (81 percent of the impact 
sample) scored an average of 5.2 scale score points higher in reading (ES = .15) 
or 2.9 months of additional learning if they were offered a scholarship compared 
to not being offered a scholarship and 6.0 scale score points higher (ES = .17) or 
3.3 months of additional learning if they used their scholarship compared to not 
being offered a scholarship. 

o Students from the first cohort of applicants (21 percent of the impact sample) 
scored an average of 8.7 scale score points higher in reading (ES = .31) if they 
were offered a scholarship compared to not being offered a scholarship and 1 1.7 
scale score points higher (ES = .42) if they used their scholarship compared to 
not being offered a scholarship. These impacts translate into 14.1 and 18.9 
months of additional learning (1.5 to 2 years of typical schooling). The statistical 
significance of this finding was not robust to adjustments for multiple 
comparisons. 
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• The OSP had no statistically significant reading impacts for other subgroups of 
participating students, including those in the lower third of the test-score performance 
distribution at baseline, boys, secondary students, and students from the second cohort 
of applicants (ES ranging from -.00 to . 1 1). 

• The OSP had no statistically significant math impacts for any of the 10 subgroups (ES 
ranging from -.16 to .23). 

Subgroup Safety and Satisfaction Impacts 

• All of the 10 subgroups analyzed, including parents of the high-priority subgroup of 
students who had attended SINI schools at baseline, reported viewing their child’s 
school as safer and more orderly if the child was offered or using an OSP scholarship 
compared to not being offered a scholarship. Effect sizes for the impact of an offer of a 
scholarship on parent perceptions of safety and an orderly school climate for the 10 
subgroups ranged from .27 to .40. Adjustments for multiple comparisons indicate that 
these 1 0 subgroup impacts on parental perceptions of safety and school climate are not 
likely to be false discoveries. 

• Consistent with the finding for students overall, none of the subgroups of students 
reported experiencing differences in safety and an orderly school climate if they were 
offered (ES range from -.03 to .08) or using an OSP scholarship. 

• In addition to an overall impact on parental satisfaction with their child’s school, the 
Program produced satisfaction impacts on 7 of the 1 0 subgroups analyzed. Effect sizes 
for the impact of an offer of a scholarship on the likelihood of a parent grading his/her 
child’s school “A” or “B” for these seven subgroups ranged from .16 to .41. 
Adjustments for multiple comparisons indicated that none of these parent satisfaction 
subgroup impacts may have been a false discovery. The parents of students who had 
attended SINI schools, parents of students in the lower one -third of the test score 
distribution, and parents of high school students generally did not report higher levels 
of school satisfaction that were statistically significant as a result of the treatment (ES 
ranged from -.03 to .13). 

• There were no statistically significant differences between the treatment group and the 
control group for all 10 subgroups in the likelihood that students gave their school a 
grade of A or B (ES ranged from -. 1 8 to .05). 



The Impact of the Program on Intermediate Outcomes 

Understanding the mechanisms through which the OSP does or does not affect student 
outcomes requires examining the expectations, experiences, and educational environments made possible 
by Program participation. The analysis here estimates the impact of the Program on a set of “intermediate 
outcomes” that may be infiuenced by parents’ choice of whether to use an OSP scholarship and where to 
use it, but are not end outcomes themselves. The method used to estimate the impacts on intermediate 
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outcomes is identical to that used to estimate impacts on the key Program outcomes, such as academic 
achievement. 



Prior to data analysis, possible intermediate outcomes of the OSP were selected based on 
existing research and theory regarding scholarship programs and educational achievement. Because 24 
intermediate outcome candidates were identified through this process, the variables were organized into 
four conceptual groups or clusters, as described below, to aid in the analysis. 

There is no way to rigorously evaluate the linkages between the intermediate outcomes and 
achievement — students are not randomly assigned to the experience of various educational conditions and 
programs. That is why any findings from this element of the study do not suggest that we have learned 
what specific factors “caused” any observed test score impacts, only that certain factors emerge from the 
analysis as possible candidates for mediating influence because the Program affected students’ experience 
of these factors. The analyses are exploratory, and, given the number of factors analyzed, some of the 
statistically significant findings may be “false discoveries” (due to chance). 



Overall, 3 years after applying to the Program, the offer of an Opportunity Scholarship 
appears to have had an impact on 8 of the 24 intermediate outcomes examined, 7 of which remained 
statistically significant after adjustments for multiple comparisons: 



• Home Educational Supports. Of the four intermediate outcomes in this category, the 
offer of a scholarship had an impact on one of them. There was a significant negative 
impact on tutor usage outside of school (ES = -.14), and this impact remained 
statistically significant after adjustments for multiple comparisons. There were no 
statistically significant differences between the treatment and control groups on 
parents’ reports of their involvement in school in year 3 (ES = -.11), parents' 
aspirations for how far in school their children would go (ES = .02), or time required 
for the student to get to school (odds ratio = 1.13).’^ 

• Student Motivation and Engagement. Of the six intermediate outcomes in this 
category, the offer of a scholarship may have had an impact on one of them. Based on 
student surveys, the offer of a scholarship seems to have had a significant negative 
impact on whether students read for frm (ES = -.16). Adjustments for multiple 
comparisons, however, indicate that this result could be a false discovery, so it should 
be interpreted with caution. There were no statistically significant differences between 
the treatment and control groups in their reported aspirations for future schooling 
(ES = -.14), engagement in extracurricular activities (ES = .04), and frequency of 
doing homework (ES = .08), or in their parents’ reports of student attendance 
(odds ratio = 1.1 1) or tardiness rates (odds ratio = 1.19). 



The effect size for this categorical variable is expressed as an odds ratio, which describes the extent to which being in the treatment group 
increases (if above 1.0) or decreases (if below 1.0) the likelihood of giving a higher-category response. 




• Instructional Characteristics. The offer of a scholarship had a statistically significant 
impact on 5 of the 10 intermediate outcomes in this group of indicators. Students 
offered a scholarship experienced a lower likelihood that their school offered tutoring 
(ES = -.38), special programs for children who were English language learners 
(ES = -.61), or special programs for students with learning problems (ES = -.36) 
compared to control group students; these impacts remained statistically significant 
after adjustments for multiple comparisons. Students offered a scholarship experienced 
a higher likelihood that their school offered programs for advanced learners (ES = .27) 
and such enrichment programs as art, music, and foreign language (ES = .23); these 
two impact estimates also remained statistically significant after adjustments for 
multiple comparisons. There were no significant differences between the treatment and 
control groups in student/teacher ratio (ES = .01), how students rated their teacher’s 
attitude (ES = -.04), the school’s use of ability grouping (ES = .02), in-school tutor 
usage (ES = .04), or the availability of before- and after-school programs (ES = -.11). 

• School Environment. The offer of a scholarship affected one of four measures of 
school environment. Students offered a scholarship experienced schools that were 
smaller by an average of 182 students (ES = -.29) than the schools attended by students 
in the control group; this impact remained statistically significant after adjustments for 
multiple comparisons. There were no statistically significant differences between the 
treatment and control groups, on average, in school reports of parent/school 
communication practices (ES = -.06), the percentage of minority students at the school 
(ES = -.10), or the classroom behavior of peers (ES = .09) based on student reports. 

It is important to note that the findings regarding the impacts of the OSP reflect the 
particular Program elements that evolved from the law passed by Congress and the characteristics of 
students, families, and schools — public and private — that exist in the Nation’s capital. The same program 
implemented in another city could yield different results, and a scholarship program in Washington, DC, 
with different design features than the OSP might also produce different outcomes. 



XX 




References 



Boruch, Robert, Dorothy de Moya, and Brooke Snyder. “The Importance of Randomized Field Trials in 
Education and Related Areas.” Evidence Matters: Randomized Trials in Education Research, 
Frederick Mosteller and Robert Boruch, editors. Washington, DC: The Brookings Institution 
Press, 2002. 

Greene, Jay P. “Vouchers in Charlotte.” Education Matters 2001, 1(2): 55-60. 

Hanushek, Eric A., John F. Kain, and Steven G. Rivkin. “Disruption Versus Tiebout Improvement: The 
Costs and Benefits of Switching Schools.” Journal of Public Economics 2004, 88: 1721-1746. 

Howell, William G., Patrick J. Wolf, David E. Campbell, and Paul E. Peterson. “School Vouchers and 
Academic Performance: Results from Three Randomized Field Trials.” Journal of Policy 
Analysis and Management 2002, 21(2): 191-217. 

Mayer, Daniel P., Paul E. Peterson, David E. Myers, Christina Clark Tuttle, and William G. Howell. 
School Choice in New York City After Three Years: An Evaluation of the School Choice 
Scholarships Program. MPR Reference No. 8404-045. Cambridge, MA: Mathematica Policy 
Research, 2002. 

Wolf, Patrick, Babette Gutmann, Nada Eissa, Michael Puma, and Marsha Silverberg. Evaluation of the 
DC Opportunity Scholarship Program: First Year Report on Participation. U.S. Department of 
Education, National Center for Education Evaluation and Regional Assistance. Washington, DC: 
U.S. Government Printing Office, 2005. Available online at [http://ies.ed.gov/ncee/]. 

Wolf, Patrick, Babette Gutmann, Michael Puma, and Marsha Silverberg. Evaluation of the DC 
Opportunity Scholarship Program: Second Year Report on Participation. U.S. Department of 
Education, National Center for Education Evaluation and Regional Assistance, NCEE 2006-4003. 
Washington, DC: U.S. Government Printing Office, 2006. Available online at 

[http ://ies. ed.gov/ncee/] . 

Wolf, Patrick, Babette Gutmann, Michael Puma, Eou Rizzo, Nada Eissa, and Marsha Silverberg. 
Evaluation of the DC Opportunity Scholarship Program: Impacts After One Year. U.S. 
Department of Education, National Center for Education Evaluation and Regional Assistance, 
NCEE 2007-4009. Washington, DC: U.S. Government Printing Office, 2007. Available online at 
[http://ies.ed.gov/ncee/]. 

Wolf, Patrick, Babette Gutmann, Michael Puma, Brian Kisida, Eou Rizzo, and Nada Eissa. Evaluation of 
the DC Opportunity Scholarship Program: Impacts After Two Years. (NCEE 2000-4023). U.S. 
Department of Education, National Center for Education Evaluation and Regional Assistance, 
NCEE 2008-4023. Washington, DC: U.S. Government Printing Office, 2008. Available online at 
[http ://ies. ed.gov/ncee/] . 



XXI 








www.ed.gov 



ies.ed.gov 




